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ABSTRACT 
The  Use  of  Judgmental  Data  in  Roll  Call  Analysis 

Despite  the  fact  that  the  roll  call  voting  patterns  of  legislators  are  being 
used  more  frequently  in  the  political  arena,  the  statistical  methods  employed 
by  political  scientists  for  analyzing  roll  calls  (e.g.  ,  Guttman  scaling, 
factor  analysis)  are  not  being  used.    The  main  reason  is  the  failure  to  incor- 
porate in  a  systematic  manner  the  subjective  estimates  of  the  political 
organizations  and  individuals  who  must  use  the  output  of  such  analyses. 
This  paper  presents  two  methods,  paired  comparison  and  constant  sum,  which 
use  judgmental  data  in  assessing  roll  calls.    Using  a  set  of  seven  defense 
policy  roll  calls  from  the  19  76  US  Senate,  the  methods  are  described  and 
compared  using  two  sets  of  judges. 


THE   USE   OF  TUDGMENTAL   DATA  IN  ROLL  CALL  ANALYSIS 

Introduction 

The  political  science  discipline  is  undergoing  an  obvious  change  in 
he  direction  of  policy  relevance.    Conferences  and  panels  increasingly  focus 
>n  the  utility  of  much  of  what  we  do  for  public  policymakers .    New  journals 
;xplicitly  focusing  on  public  policy  have  emerged.     Textbooks  on  American 
fovernment  now  must  have  "public  policy"  somewhere  in  the  title.    Government 
unding  for  research  is  increasingly  contingent  on  the  political  scientist  demon- 
strating the  payoff  of  the  proposed  research  for  policymaking. 

One  area  in  political  science  where  this  does  not  appear  to  be  the  case 
s  legislative  roll  call  analysis.    It  is  the  purpose  of  this  article:    1)  to  first 
show  that  the  use  of  roll  calls  in  politics  is  increasing;  2)  to  demonstrate  why 
most  of  the  research  in  roll  call  analysis  has  not  been  utilized  i'or  policymaking; 
3)  and  put  forth  several  methods  for  using  judgmental  data  in  assessing  roll  calls 
which  will  make  this  type  of  analysis  more  useful  in  the  policy  arena. 
Increased  Importance  of  Voting  In  The  Political  Arena 

The  use  of  roll  calls  to  describe  the  performance  of  Congressmen  in 
general  and  on  specific  issues  is  not  new.    For  example,  the  American  Federation 
Df  Government  Employees  has  rated  Congressmen  since  the  1940s,  and  the 
\mericans  for  Democratic  Action  since  1948.    But  it  wasn't  until  the  early  19  70s 
that  the  "rating  game"  swung  into  high  gear  and  became  an  important  factor  in 


election  campaigns  .    The  key  was  the  shift  from  the  general  orientation  approach 
(ADA,  Congressional  Quarterly's  Conservative  Coalition,  etc.)  to  one  focused 
on  specific  issues. 

No  better  example  could  be  given  than  the  impact  of  the  Environmental 
Action's  Dirty  Dozen  campaign.     In  1970,  1972,  1974  and  1976  this  group  used 
roll  call  analysis  to  identify  and  publicize  the  twelve  Congressmen  with  the 
"worst"  anti-environment  record.    Through  1974,  31  different  Congressmen  were 
so  identified.    Seventeen  were  defeated  in  the  first  year  being  listed.    Five 
were  defeated  in  subsequent  elections.    Two  retired  and  seven  remain  in 
Congress.      In  1976,  three  of  the  "dirty  dozen"  were  defeated. 

This  use  of  roll  calls  to  pinpoint  Congressmen  who  are  performing 
"poorly"  or  not  in  the  best  interests  of  certain  groups  is  not  limited  to  the  liberal 
causes  such  as  consumerism  and  environment.    For  example,  in  November  1975 
the  Atlantic  Richfield  Company's  Civic  Action  Program  (CAP)  compiled  legislator 
scores  on  the  energy  industry's  position  based  on  six  key  energy  votes.     The 

clearly  political  intent  is  obvious  since  they  listed  only  legislators  "who  rep- 

2 
resent  a  substantial  number  of  CAP  members.  "      They  singled  out  Rep.  Robert 

W.  Edgar  (D-PA)  for  special  attention,  noting  that  his 

votes  since  election  can  be  characterized  as  being  in  opposition 
to  the  search  for  and  production  of  additional  domestic  energy 
supplies  .  .  .  .As  a  freshman  in  a  competitive  political  situation 
and  as  a  newcomer  to  the  complexities  of  the  energy  industry, 
he  is  a  prime  candidate  for  factual  information.  3 
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Perhaps  the  most  bizarre  example  of  the  use  of  roll  calls  In  the  1976 
lection  campaign  was  the  ratings  compiled  by  the  National  Alliance  of  Senior 
Citizens  (NASC)  and  the  National  Council  of  Senior  Citizens  (NCSC) .    The  NASC 
i"Golden  Age  Index"  rates  Congressmen  based  on  votes  "perceived  most  demon- 
strative of  a  commitment  to  fiscal  responsibility.         Republicans  scored  exceed- 
.ngly  high  in  this  index.    The  NCSC  index,  on  the  other  hand,  is  concerned  more 
with  governmental  programs  in  support  of  the  elderly,  such  as  pension  reform 
and  Medicare  benefits.     Not  surprisingly,  high  scores  on  this  index  rarely 
coincide  with  high  scores  on  the  NASC  index.     In  one  heated  California  debate 
between  two  candidates,  a  liberal  incumbent  was  accused  of  "not  caring  about 
the  elderly"  based  on  his  NASC  (fiscal  responsibility)  score.    Although  he  was 
quick  to  point  out  his  high  score  on  the  NCSC  index  and  secure  an  apology  from 
his  opponent,  the  incident  points  out  not  only  the  increased  use  of  roll  call 
analysis  in  politics  but  also  the  problems  which  occur  when  poorly  constructed, 
invalid  and  .misleading  indices  are  essentially  misused. 

What  is  required  is  some  sort  of  quality  control  of  the  "rating  game.  " 
A  good  example  of  suggested  control  is  a  May  19  76  study  entitled  The  Rating  Game, 
published  by  the  House  Republican  Research  Committee.    This  report  recommends 
a  "code  of  ethics  , "  several  parts  of  which  are  relevant  to  the  problems  of  roll 
call  analysis  addressed  in  this  article.    Their  code  includes 

an  objective,  rational  and  understandable  vote  selection 
process,  a  guarantee  of   membership   input  in  the  determina- 
tion of  the  rating  group's  concerns,  detailed  vote  descrip- 
tions, avoidance  of  percentage  compilations,  and  tech- 
nical accuracy  in  description  and  tabulations  .  5 
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Political  scientists  may  legitimately  ask  to  what  extent  the  scholarly  work  on 
roll  call  analysis  has  addressed  and  perhaps  alleviated  the  problems  cited. 
It  is  to  such  an  inquiry  that  we  now  turn,  examining  briefly  what  has  been 
done  and  its  applicability  to  the  policymaking  arena. 

Roll  Call  Analysis  in  Political  Science 

No  attempt  will  be  made  to  duplicate  the  several  textbooks  and  articles 
which  adequately  survey  the  field  of  roll  call  analysis.     However,  a  brief  sum- 
mary of  some  of  the  more  commonly  used  methods  will  shed  some  light  on  the 
general  problem  of  using  this  type  of  analysis  for  policymaking. 

By  far  the  most  frequently  used  method  of  using  roll  calls  to  describe 
and  explain  legislative  behavior  is  the  simple  percentage  technique.    Specifi- 
cally, the  various  Congressional  Quarterly  indices  are  the  data  used  in  many  of 
the  major  studies  of  legislative  behavior.       This  method  assumes  that  each  roll 
call  is  equally  weighted  and  is  an  attractive  technique  since  the  simple  percen- 
tage is  a  ratio  measure  which  facilitates  comparison  across  legislators  and  time, 
and  can  be  used  in  a  variety  of  statistical  techniques. 

g 

Another  popular  method  is  Guttman  scaling,    especially  since  the  advent 
of  statistical  computer  packages  which  facilitate  construction  of  the  scales  .     It 
has  an  advantage  over  simple  percentage  scales  in  that  it  empirically  checks  for 
unidimensionality  of  the  roll  calls  selected.    A  drawback  of  this  method  is  that 
it  produces  only  an  ordinal  scale.    This  has  not  stopped  many  a  political  scien- 
tist from  assuming  interval  data  and  proceeding  to  invalidly  use  such  scales  in 

q 
regression  analysis  .  J 
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One  of  the  disadvantages  of  the  Guttman  scaling  approach  is  the  lim- 
ited number  of  roll  calls  which  can  be  used  to  construct  a  scale  (SPSS  limits 
the  analyst  to  12).    Factor  analysis    is  a  third  method  of  constructing  scales 
which  alleviates  this  problem.    As  with  the  two  previous  methods,  each  roll 
sail  is  equally  weighted.    The  ability  to  process  a  greater  number  of  roll  calls 
makes  the  selection  and  grouping  of  roll  calls  by  issue-area  somewhat  more 
objective  than  the  Guttman  technique,  where  some  sort  of  subjective  sorting 
process  must  be  used  in  order  to  get  the  number  of  roll  calls  to  a  workable  size. 
It  should  be  added  here  that  a  significant  amount  of  academic  work  has  been 
done  comparing  the  statistical  properties  and  structures  of  Guttman  scale  and 
factor  solutions.      Warwick's  evaluation  of  the  two  techniques  concludes  that 
factor  analysis  is 

a  refinement  over  Guttman  scaling  in  the  assessment  of  roll 
call  similarity,  although  where  this  extra  precision  is  not 
required,  Guttman  scaling  remains  an  acceptable  methodology. 

There  have  been  several  other  methods  of  roll  call  analysis  which  have 

been  developed.    Specifically,  multi-dimensional  scaling,      cluster-bloc 

14  15 

analysis      and    smallest   space  analysis.         Without  judging  their  future 

utility  in  describing  or  explaining  legislative  behavior,  it  can  be  said  that 

they  have  not  been  used  to  any  great  extent  to  this  point.    In  addition,  the  three 

methods  previously  summarized  will  serve  to  make  the  next  point  in  this  article, 

namely  that  despite  their  statistical  rigor,  these  methods  fall  short  as  techniques 

which  can  be  utilized  in  policymaking. 
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What  are  the  shortcomings  of  these  methods?    First,  and  most  crucial 
to  politics,  is  the  selection  of  issues  to  make  up  the  indices  and  scales.     In 
1976,  there  were  688  roll  call  votes  in  the  Senate.    If  the  policy  problem  is  to 
determine  where  a  senator  stands  in  regard  to  defense  policy,  which  votes  are 
selected?    To  use  Guttman  scaling,  a  small  subset  of  votes  must  be  selected. 
As  Warwick  points  out,  Guttman  scaling  is  extremely  sensitive  to  the  vote 
splits  in  each  roll  call.        Out  of  30  defense  roll  calls,  10  may  scale,  10  may 
not,  simply  based  on  the  Yes-No  breakdown  on  the  vote.     Factor  analysis 
improves  the  selection  process  somewhat,  in  that  it  can  handle  a  larger  number 
of  roll  calls  (although  not  the  688  roll  calls  in  1976).     In  my  dissertation  research 
I  found  that  factor  analysis  is  more  discriminating  than  Guttman  scaling  (e.g. 

some  military  assistance  roll  calls  appeared  in  the  Guttman  scales  but  formed 

1 7 
a  distinct  cluster  when  factored.)        However,  the  problem  which  still  remains 

is  insuring  that,  on  any  given  roll  call,  a  "Yes"  is  pro — or  anti-defense.    Was 

the  vote  a  procedural  protest  of  some  sort?    Clearly,  both  Guttman  and  factor 

analysis  cannot  address  this  problem. 

A  second  problem  with  the  three  commonly  used  methods  of  roll-call 

analysis  is  the  assumption  that  each  roll  call  adds  equally  to  the  dimension 

being  measured.    Some  votes  are  seminal,  others  on  the  same  issue  are  trivial. 

While  these  techniques  can,  over  a  large  set  of  roll  calls,  give  the  general 

orientation  of  a  legislator,  this  is  not  adequate  for  policy  purposes.    This  is 

particularly  noticeable  in  the  defense  issue-area.    Senators  Cranston  and  Tunney 

would  come  out  in  any  general  evaluation  as  somewhat  critical  of  the  Defense 
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Department.    Yet  on  the  several  roll  calls  to  cut  the  B-l  bomber,  they  were  very 

supportive  of  continuing  the  program. 

There  has  been  one  recent  attempt  to  solve  this  problem  of  determining 

the  relative  importance  of  roll  calls.     Using  the  House  of  Representatives  as 

his  source  of  data,  Frans  Bax  has  proposed  an  Index  of  Importance  which 

relies  upon  the  practice  of  the  House  to  allocate  varying 
amounts  of  time  for  debate  on  legislation  coming  to  the 
floor.  .  .  .Bills  receiving  a  rule  are  given  a  value  on  the 
Index  equal  to  the  number  of  hours  granted  for  a  debate. 

Bax  then  takes  his  index  and  finds  that  it  correlates  it  with  those  attributes  of 

legislation  that  suggest  importance — occurrence  of  a  roll  call  vote  on  the  bill, 

frequency  of  roll  call  votes  on  the  bill,  partisan  controversy,  controversy  within 

committee,  number  of  legislators  voting,  Presidential  involvement,  and  floor 

action  in  the  form  of  amendments  to  the  bill.        Bax  concludes  that  his  "Index  is 

closely  and  explicitly  linked  to  the  judgments  of  those  House  members  who  are 

20 
expected  to  assess  the  relative  importance  of  many  bills  . " 

Bax  has  overcome  some  of  the  obstacles  to  using  roll  calls  in  policy 
analysis,  particularly  the  problem  of  constructing  unobtrusive  measures  of 
importance.    But  his  goal  is  similar  to  other  approaches  in  that  he  is  attempting 
to  create  a  picture  of  how  the  House  "really"  feels  about  the  relative  importance 
of  roll  calls.     His  Index  of  Importance  is  an  important  step  forward  in  under- 
standing the  House,  but  does  not  address  the  concern  of  this  article.     Namely, 
how  can  we  evaluate  the  interaction  of  the  Congress  with  outside  groups  (interest 
groups,  executive  agencies,  etc.)  using  roll  call  analysis?    What  the  House 
deems  important  in  defense  policy  (e.g.  lengthy  debate  on  the  neutron  bomb  in 
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the  summer  of  1977)  may  not  be  as  important  to  the  Defense  Department,  or 
specific  groups  within  DoD  such  as  the  Navy. 

The  missing  ingredient  in  all  the  approaches  is  the  expert  judgment  which 
must  go  into  both  the  selection  and  weighting  process.    In  terms  of  selection, 
most  organizations  constructing  voting  indices  do  have  a  panel  which  selects 
votes  but  rarely  do  they  specify  why  the  votes  were  selected,  and  why  they 
are  a  good  indicator  of  the  dimension  being  measured.     This  can  only  lead  to 
increased  suspicion  that  votes  are  being  selected  to  create  a  pattern  for  political 
gain,  a  suspicion  which  leads  to  little  use  of  the  index.    Although  somewhat 
general  for  most  policy  purposes,  the  ADA  index  has  been  used  consistently 
for  many  years  as  a  credible  index  of  a  liberal  voting  record.    As  far  as  weighting 
is  concerned,  little  or  none  is  conducted.     I  think  the  main  reason  for  this  is 
that  in  their  quest  for  objectivity,  the  raters  feel  that  distinguishing  the  impor- 
tance of  roll  calls  is  blatantly  subjective,  and  hence  don't  even  attempt  it.    The 
major  problem  is  that  the  search  for  objectivity  is  a  futile  one.    A  legislator 
does  not  possess  one  "objective"  attitude  toward  environmental,  defense  or 
farm  issues.     Rather,  there  are  various  versions  of  this  attitude  which  vary 
according  to  the  perceptions  of  the  conflicting  groups  which  determine  policy  in 
accordance  with  what  Allison  has  termed  bureaucratic  politics.     If  this  is  how 
policy  is  made,  and  a  legislator's  voting  record  is  to  be  used  as  a  measure  of 
previous  stances  on  issue-areas,  it  is  obvious  that  attention  must  be  paid  to  how 
each  group  views  the  voting  record.    This  means  systematically  tapping  expert 
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opinion  regarding  which  votes  make  up  an  issue-area,  and  their  relative  weight 
In  contributing  to  the  dimension  being  measured.  It  is  to  this  task  that  we  now 
turn. 

The  Use  of  Judgment  Data  In  Roll  Call  Analysis 

Having  concluded  that  expert  opinion  must  be  systematically  measured 
if  roll  call  analysis  is  to  be  used  in  the  political  process,  we  now  address  spec- 
ifically how  this  can  be  accomplished.    The  two  basic  steps  involve  the  selection 
of  votes  for  a  particular  issue-area  or  dimension,  and  the  weighting  of  those 
votes  in  terms  of  their  importance  to  the  group  making  the  index  or  scale.     This 
article  is  more  concerned  with  the  second  step,  so  the  first  step  of  vote  selection 
will  be  only  briefly  mentioned.     Obviously  some  sort  of  Delphi  technique  could 
be  used  to  arrive  at  a  consensus  as  to  what  votes  make  up  the  dimension  in 
question.    If  such  a  selection  is  destined  to  be  the  rating  organization's  view, 
care  must  be  taken  to  insure  that  the  judges  selected  to  evaluate  roll  calls  in 
some  way  represent  the  organization  and  produce  a  true  "party  line.  "    How  this 
is  done  is  much  more  a  matter  of  organizational  behavior  and  beyond  the  scope  of 
this  article. 

But  once  agreement  is  reached  on  what  roll  calls  make  up  a  certain 
dimension,  some  sort  of  weighting  procedure  must  be  used  so  that  a  valid  picture 
of  a  legislator's  record  is  constructed.    Two  such  procedures  were  used  in  this 
study . 
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The  Method  of  Paired  Comparisons 
The  basic  idea  in  this  method  is  that  a  judge  can  rank  any  two  roll  calls 
as  to  their  relative  importance.    If  the  scale  being  constructed  contains  n  roll 
calls,  each  judge  will  make  n(n-l)/2  comparisons.    The  paired  comparison 
method  converts  these  ordinal  judgments  into  an  interval  scale.    The  method  is 

not  new,  having  been  in  the  methodological  kits  of  psychologists  since  Thurstone 

21 

spelled  out  the  Law  of  Comparative  Judgement  in  192  7.        There  have  been  many 

22 
applications  of  this  method  in  political  science       but  none  which  address  roll 

call  voting.    An  example  from  the  several  experiments  conducted  during  this 
research  will  illustrate  the  method. 

Seven  Senate  roll  calls  on  defense  policy  were  selected  by  the  rating 
organization  (in  this  case,  I  selected  them  for  experimental  purposes)  as  indica- 
tors of  general  attitude  toward  the  Defense  Department.    The  judges  were  given 
a  brief  description  and  background  on  the  vote,  along  with  the  recorded  results 
of  the  vote.    They  were  instructed  to  assume  the  role  of  Secretary  of  Defense  in 
1976,  and  then  rank  the  votes  in  terms  of  their  importance  to  you  as  Secretary  of 
Defense.    No  ties  were  permitted.    Two  groups  of  judges  were  used.     One  was 
my  class  in  data  analysis  which  had  taken  two  courses,  one  in  international 
relations  and  another  in  American  national  security  policy  within  the  last  six 
months  (hereafter  referred  to  as  Group  NAVAL) .    The  other  group  was  made  up  of 
participants  in  the  National  Security  Education  Seminar  held  each  summer  at 
Colorado  College,  a  group  with  considerably  more  expertise  and  knowledge  in 
national  security  affairs  (referred  to  as  Group  NSES). 
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They  were  given  a  sheet  of  paper  in  the  following  format: 
Most  Important 


Least  Important 


This  raw  data  was  then  transformed  into  a  matrix  as  shown  in  Table  1. 
The  data  can  be  interpreted  as  follows.    There  were  4  out  of  6  judges  who  felt 
yote  #12  was  more  important  than  #6,  while  only  1  judge  thought  #9  was  more 
.mportant  than  #6 . 

These  raw  frequency  rankings  are  next  transformed  into  an  array  of  pro- 
portions,  in  which  each  cell  entry  (Pj^)  is  equal  to  the  proportion  of  times  roll 
call  k  was  judged  more  important  than  roll  call  j.     (See  Table  2) 

The  next  step  involves  transforming  these  proportions  into  Z  scores 
using  the  table  of  normal  distribution.     The  cell  entries  (Zj^)  now  equate  to  the 
unit  normal  deviate  corresponding  to  the  proportions  in  Table  2.     (See  Table  3) 

The  final  step  in  the  process  is  the  computation  of  a  scale  score  for 
each  roll  call,  and  is  also  included  in  Table  3  above.    As  can  be  seen,  the 
scale  score  for  each  roll  call  is  the  mean  of  each  column.    While  the  reader  is 
referred  to  Torgerson  ^  for  the  theory  and  assumptions  implicit  in  the  technique, 
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Table  1 
Raw  Data  Matrix  For  Paired  Comparison  Technique 

Roll  Call  #  (k) 


12 


13 


15 


16 


(j)  12 


13 


15 


16 


- 

1 

1 

4 

1 

2 

1 

5 

- 

3 

4 

1 

3 

4 

5 

3 

- 

4 

1 

4 

2 

2 

2 

2 

- 

2 

2 

2 

5 

5 

5 

4 

- 

4 

4 

4 

3 

2 

4 

2 

- 

3 

5 

2 

4 

4 

2 

3 

- 
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Table  2 
Proportion  Matrix  For  Paired  Comparison  Technique 

Roll  Call  (k) 


12 


13 


15 


16 


Roll 

Call 

(j) 


12 


1'3 


15 


16 


- 

.20 

.2 

.667 

.2 

.333 

.2 

.833 

- 

.5 

.667 

.2 

.5 

.667 

.833 

.5 

- 

.667 

.2 

.667 

.333 

.333 

.333 

.333 

- 

.333 

.333 

.333 

.833 

.833 

.833 

.667 

- 

.667 

.667 

.667 

.5 

.333 

.667 

.333 

- 

.5 

.833 

.333 

.667 

.667 

.333 

.5 

- 
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Table  3 


Roll 

Call 

(j) 


n 


Z  Score  Matrix  For  Paired  Comparison  Technique 

Roll  Call  (k) 


12 


13 


15 


16 


1 

fi 


Zjk 


12 


13 


15 


16 


- 

-.841 

-.841 

.43 

-.841 

-.43 

-.841 

.965 

- 

0 

.43 

-.841 

0 

.43 

.965 

0 

- 

.43 

-.841 

.43 

-.43 

-.43 

-.43 

-.43 

- 

-.43 

-.43 

-.43 

.965 

.965 

.965 

.43 

- 

.43 

.43 

.43 

0 

-.43 

.43 

-.43 

- 

0 

.965 

-.43 

.43 

.43 

-.43 

0 

- 

.5514 

-.1051 

-.0437 

.3685 

-.5447' 

0 

-.1201 
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we  can  and  should  look  at  what  has  been  done  here.    The  best  way  to  do  this 
is  look  at  the  highest  and  lowest  scores .    The  roll  call  with  the  highest  score 
is  #6.    It  received  that  score  due  to  the  basic  fact  that  in  most  cases  the  judges 
ranked  it  above  all  others.    Roll  call  #13  received  the  lowest  score  because  in 
most  cases  it  ranked  below  the  other  roll  calls.    Roll  call  #15  is  in  the  middle  of 
the  scale  since  overall  the  judges  were  ambivalent  concerning  its  importance. 

The  resulting  interval  scale  can  be  transformed  to  make  it  more  inter- 
retable.     In  this  case,   .5447  was  added  to  each  scale  score  resulting  in  the 
following  scale  of  importance  for  the  7  defense  policy  roll  calls.     (See  Table  4) 

Since  these  now  are  interval  data,  some  basic  conclusions  can  be 
make  relative  to  the  importance  of  the  roll  calls  .     Mainly,  there  appears  to  be 
three  levels  of  importance--6  and  12,  15,  9,   7,  and  16,  and  13.    The  problem  is 
that  very  little  else  can  be  done  with  the  scale.    The  purpose  behind  weighting 
the  votes  was  to  apply  them  to  individual  legislators  so  that  a  "defense"  score 
could  be  produced.    Since  these  are  interval  and  not  ratio  data  (i.e.  ,  the  0 
point  is  not  meaningful  since  it  does  not  indicate  an  absence  of  the  dimension), 
one  cannot  modify  (multiply)  a  'Yes  '  vote  by  the  weight  of  the  roll  call.    An 
illustration  makes  the  point.     Listed  in  Table  5  are  the  results  of  our  experiment 
so  far.    Scale  A  is  the  original  scale,  scale  B  has  added  +1  to  each  value.    The 
intervals  in  both  scales  are  identical.    Now  we  apply  both  scales  to  three 
hypothetical  voting  records. 
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Table  4 
Defense  Support  Scale,  Paired  Comparison  Technique 

Roll  Call  Scale 

#  Subject  of  Roll  Call Score 

6  Cut  off  funds  for  B-l  1.0961 

12  Conference  Report  -  Defense  Procurement  .9132 

15  Cut  funds  for  Lance  missile  .5447 
9                 Cut  off  funds  for  Minuteman  -  Auth  .501 

7  Delay  B-l  decision  until  Feb  77  .4396 

16  Cut  off  funds  for  nuclear  carrier  .4246 

13  Defer  decision  on  Minuteman  .00 


Table  5 
Adjusted  Defense  Support  Scale,  Paired  Comparison  Technique 

Roll  Call 

6 
12 
15 

9 

7 

16 
13 


Scale  A 

Scale  B 

1.0961 

2.0961 

.9132 

1.9132 

.5447 

1.5447 

.5010 

1.5010 

.4396 

1.4396 

.4246 

1.4246 

.00 

1.00 

-16- 


(+)  =  pro-defense  vote 

6    12   15    9     7    16   13  Score  A  Score  B 

Cong.  X+    +     -     +     +     -     +  2.9499  7.9499 

Cong.  Y++----+  2.0093  5.0093 

Cong.  Z----    +    --  .4396  1.4396 

To  see  what  has  happened  here,  we  compare  the  intervals  between 

Congressmen  X  and  Y,  and  Y  and  Z,  for  both  scales  A  and  B.    They  should  be 

proportional.    But  they  are  not. 

(2.9499-2.0093)-/  (7.9499-5.0093)  .      .9406    /  2.9406  .        5992^8237 
(2.0093-.4396)        (5  .  0093-1.  4396)  '    1 .  5697      3.5697' 

While  the  paired  comparison  method  can  shed  some  light  on  the  relative 
importance  of  roll  calls  to  a  group  of  judges  ,  the  resulting  interval  scale  cannot 
be  used  for  the  ultimate  purpose  of  rating  legislators  on  a  specific  dimension. 

The  Constant  Sum  Method 

The  key  to  the  solution  of  the  above  problem   is  to  obtain  ratio  data  from 
those  judging  the  roll  calls.    Several  alternatives  exist.    The  first  is  to  tell  the 
judges  to  give  each  roll  call  a  value  from  0  to  1.0  such  that  the  final  results 

reflect  the  ratios  among  roll  calls.    If  one  roll  call  is   .4  and  another  .8,  the 

°4 
latter  contains  twice  as  much  of  the  property  (e.g.  anti-defense)  as  the  former.^ 

Experience  with  this  technique  has  shown  that  it  is  very  difficult  for  judges  to 

think  in  ratio  terms,  especially  with  more  than  a  few  cases  to  evaluate.    °    This 

7  fi 

has  led  to  the  so-called  "pie  allocation"  approach,         in  which  judges  are  given 
a  pie  of  100  points,  and  asked  to  split  it  up  so  that  each  piece  not  only  has  a 
size  relative  to  all  others,  but  the  graphic  aspect  of  the  technique  gives  more 
assurance  that  the  judges  are  thinking  in  ratio  terms. 
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There  exists  a  third  alternative,  however,  which  also  produces  ratio 
data  but  puts  much  less  of  a  demand  on  the  judge.    This  method  is  generally 

termed  direct  estimation  of  sense  ratios,  or  more  specifically,  the  constant  sum 

27 

method.    As  with  the  paired  comparison  technique,  it  is  not  a  new  method. 

The  stimuli  (in  our  case  the  descriptions  of  the  roll  calls)  are  presented  to  the 
judges  in  pairs  .    The  judge  is  instructed  to  divide  100  points  between  them  in 
accordance  with  the  absolute  ratio  of  the  greater  to  the  lesser.    Although  the 
judge  still  must  think  in  ratio  terms  (assigning  80  points  to  one  roll  call  and  2  0 
to  the  other  indicates  the  former  is  4  times  as  "important"  as  the  latter),  his  task 
is  simplified  since  he  only  must  deal  with  one  ratio  at  a  time.    Since  the  judge 
does  have  the  opportunity  to  assign  0  points  to  a  roll  call,  a  natural  0  point  in 
fact  exists . 

How  these  raw  data  are  converted  into  scale  scores  can  best  be  describee 
by  returning  to  the  previous  set  of  roll  calls  and  judges.     Having  simply  ranked 
the  roll  calls  in  importance,  the  judges  were  then  given  the  roll  calls  in  pairs 
and  asked  to  split  100  points  between  them.    They  were  told  that  an  80-2  0  split 
between  a  B-l  issue  and  a  nuclear  carrier  issue  would  indicate  that  the  B-l  roll 
call  is  4  times  as  critical  to  DoD  as  the  nuclear  carrier  roll  call  in  determining 
whether  a  senator  is  '"pro"  or  "anti"  defense. 

The  first  matrix  produced  (Table  6)  is  a  matrix  V  in  which  each  cell  (V-jJ 
is  the  average  number  of  points  assigned  to  roll  call  k  when  compared  to  roll  call 
J. 
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Table  6 

Matrix  V  With  Elements  Denoting  The  Average  Number  Of 
Points  Assigned  To  Roll  Call  k  When  Compared  To  Roll  Call  j 

Roll  Call  k 


12 


13 


15 


16 


Roll 

Call         12 
j 

13 


15 


16 


50.0 

24.3 

27.9 

55.7 

19.3 

32.2 

50.0 

75.7 

50.0 

52.9 

74.7 

41.2 

51.4 

50.3 

72.1 

47.1 

50.0 

75.0 

22.9 

50.0 

50.8 

44.3 

25.3 

25.0 

50.0 

19.3 

27.1 

29.3 

80.7 

58.9 

67.1 

80.7 

50.0 

59.3 

67.1 

67.9 

48.6 

50.0 

72.9 

40.7 

50.0 

40.0 

50.0 

49.7 

49.3 

70.7 

32.9 

60.0 

50.0 
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The  next  step  is  to  construct  a  matrix  W  in  which  each  cell  is  the  ratio 
of  the  average  points  indicated  by  the  column  to  those  indicated  by  the  row  (Table 

7).   (wjk=Yin) 
vkj 

The  final  step  involves  calculating  the  geometric  means  of  the  columns 

?  8 
of  the  above  matrix.         The  scale  values  for  each  roll  call  (SJ  are  calculated 

using  the  following  equation: 

n 
log  Sk  =^T   lo<3  wjk 
Jzl 


n 

where  n  =  the  number  of  roll  calls . 

The  actual  scale  values  are  the  antilogs  of  the  above  values ,  calculated 
as  follows: 

Sk  = 
The  scale  value  for  roll  call  6  is  calculated  as  follows: 


77"  w* 


l/n 


S6  =  [(1)(3.118)(2.589(.794)(4.184)(2.110)(1)]1//7  =  1.78 
Us'ing  the  above  approach,  two  sets  of  scales  were  constructed;  one 
using  the  experts  from  the  National  Security  Education  Seminar  (NSES)  and  the 
other  using  naval  officers  from  my  data  analysis  class  (NAVAL).    The  following 
table  gives  the  results  of  these  two  experiments.      (See  Table  8) 

In  a  quick  test  of  the  rank  orders  of  the  four  scales  ,  a  non-parametric 
test  (Kendall's  tau)  was  conducted  to  determine  the  level  of  association  of  the 
four  scales.     (See  Table  9) 
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Table  7 

Matrix  W  With  Elements  Denoting  The  Ratio  Of  Average  Points 
Indicated  By  The  Column  To  Those  Indicated  By  The  Row 

Roll  Call  k 


12 


13 


15 


16 


Roll 

Call 

J 


12 


13 


15 


16 


1 

.321 

.386 

1.259 

.239 

.474 

1.00 

3.118 

1 

1.12 

2.958 

.699 

1.059 

1.012 

2.589 

.892 

1 

.667 

.340 

1.00 

1.028 

.794 

.338 

.333 

1 

.239 

.373 

.414 

4.184 

1.430 

2.937 

4.181 

1 

1.457 

2.044 

2.110 

.944 

1.00 

2.683 

.686 

1 

.667 

1.00 

.988 

.972 

2  .  414 

.489 

1.5 

1 
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Table  8  -  Defense  Support  Scales,  Constant  Sum  Method 
Roll  Call 
S12  -  Conference  Report  -  Defense  Procurement 

56  -  Cut  off  Funds  for  B-l 
Sig  -  Cut  off  Funds  for  Nuclear  Carrier 
Sg  -  Cut  off  Funds  for  Minute  man  -  Auth 
S,r  -  Cut  off  Funds  for  Lance  Missile 

57  -  Dalay  B-l  Decision  Unitl  February  77 
S^  -  Defer  Decision  on  Minuteman 


NSES 

NAVAL 

(n=6) 

(n=19) 

1.826 

1.664 

1.780 

1.683 

.927 

1.098 

.881 

1.082 

.880 

.688 

.746 

.736 

.463 

.594 

Table  9 

Rank  Order  Correlation  of  Guttman, 
Paired  Comparison  And  Constant  Sum  Scales 


Guttman 

NSES 

NAVAL 

Paired 
Comparison 


Guttman 


NSES 

.428 

1 


Paired 

NAVAL 

C 

omDarisoi 

.238 

.  5  u  0 

.857 

.571 

1 

.523 
1 

-22- 


The  two  constant  sum  scales  were  highly  related,  whereas  the  other 
elationships  were  much  lower.    We  have  now  produced  ratio  data  which  not  only 
iifferentiates  each  roll  call  in  terms  of  its  importance,  but  also  in  a  form  that 
:an  be  used  in  assigning  scores  (in  this  case  "anti-defense"  scores)  to  indi- 
/idual  legislators.    Some  examples  using  1976  votes  will  demonstrate  the  use  of 
the  various  techniques .     (See  Tables  10  and  11) 

In  each  of  the  tables,  the  "Guttman"  score  is  a  simple  index  of  the 
number  of  anti-defense  votes  (common  practice  in  most  Guttman  scale  analyses) . 
In  Table  10,  the  scale  scores  for  the  "pure"  types  of  senators  are  listed.    The 
first  conclusion  drawn  is  that  using  tha  rank-order  scores  produced  by  the  Guttman 
procedure  as  interval  data  would  be  seriously  misleading.    Second,  there  are  some 
significant  differences  when  Senator  Burdick  is  rated  by  the  two  groups  of 
experts  (NSES  vs  NAVAL) .    If  NAVAL  represents  either  an  interest  group  or  gov- 
ernmental bureaucracy  which  keys  their  activity  to  anti-defense  activity,  they 
will  perhaps  be  watching  Burdick  more  closely  than  NSES. 

*    The  key  conclusions,  however,  are  drawn  from  Table  11,  in  which  the 
scores  of  "mixed"  types  are  listed.     It  should  simply,  but  significantly,  be  noted 
that  senators  with  identical  Guttman  scores  have  different  NSES  and  NAVAL  scores, 
and  conversely,  similar  NSES  and  NAVAL  scores  produce  dissimilar  Guttman  scores, 
In  the  case  of  mythical  Senator  C    the  lower  Guttman  score  produces  a  higher 
NSES  and  NAVAL  score.     Clearly,  the  unidimensionality  test  implicit  in  Guttman 
scaling  is  inadequate  in  accurately  depicting  the  magnitude  of  the  "anti-defense" 
dimension  for  individual  senators. 
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Senator 

Buckley 

Nunn 

Magnuson 

Burdlck 

Hartke 

Leany 

McGovern 


12 


Table  10 

Constant  Sum  Scale  Scores  For  Pure  Type  Senators 
(+)  -  Anti-Defense  Position  on  Roll  Call 

Roll  Call  Number 

IS      6 ? 13 16 7 


Guttman        NSES      NAVAL 
Score  Score     Score 


0 

0 

0 

1 

.746 

.736 

2 

1.673 

1.834 

4 

3.017 

3.510 

5 

4.797 

5.193 

6 

5.677 

5.881 

7 

7.503 

7.545 

Senator 

Goldwater 

Wclcker 

Percy 

Randolph 

Schwelker 

Cranston 

Hatfield 

A 

B 
C 
D 


12 


Table  11 
Constant  Sum  Scale  Scores  For  Mixed  Type  Senators 
Roll  Call  Number 
15 6 9 13 16 7 


Guttman 

NSES 

NAVAL 

Score 

Score 

Score 

1 

.927 

1.098 

2 

1.344 

1.676 

3 

2.090 

2.412 

3 

2.554 

2.755 

4 

3.870 

3.964 

4 

3.151 

3.462 

6 

6.623 

6.857 

4 

4.416 

4.5S1 

4 

2.970 

3.100 

2 

3.606 

3.347 

4 

3.017 

3.510 
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Conclusions 
The  methods  put  forth  in  this  article  are  intended  to  spur  discussion  and 
esearch  regarding  how  roll  call  data  can  and  should  be  employed  in  the  policy- 
laking  arena.    It  has  been  assumed  that  the  bureaucratic  politics  model  accurately 
ixplains  policy  outputs,  that  policy  is  a  function  of  bargaining,  power  and  influence 
.egislators'  votes  are  a  record  of  their  position,  and  as  such  form  the  "previous 
stances"  useful  in  the  bargaining  which  constitutes  politics.    This  is  not  to  say 

hat  there  are  not  serious  normative  questions  concerning  the  permanent  adoption 

29 
)f  this  model,         or  that  the  search  for  an  objective  measure  using  roll  calls  is 

lot  a  desirable  goal.     Rather,  what  has  been  said  here  is  that  bureaucratic  politics 

.s  descriptive  of  how  policy  is  made,  roll  call  analysis  (however  crude)  is  part 

of  the  currency  in  such  policymaking,  and  more  attention  should  be  paid  to 

designing  methods  which  take  into  account  the  selection  and  weighting  of  roll 

calJs  by  experts  representative  of  the  organizations  involved. 
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